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ABSTRACT 


This  paper  is  a  product  of  the  Institute  for  Water  Resources  (IWR) 
research  on  uncertainty  in  benefit  analysis.  The  research  is  intended  to 
enable  Corps  of  Engineer  economists  to  make  better  estimates  of  project 
benefits  with  limited  resources,  and  to  help  the  Corps  to  better  orient  its 
research  program  to  provide  useful  methods  and  empirical  data  for  resolving 
uncertainty  in  project  planning.  To  help  focus  the  effort,  IWR  organized  a 
workshop  on  29  April  1988  to  determine  the  current  attitudes  and  practices 
from  a  survey  of  Corps  economists.  This  report  summarizes  the 
questionnaire  which  covered  the  most  significant  topics  of  that  workshop. 
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lUTRODUCTION 


In  April  1988,  Immediacely  following  Che  Corps  of  Engineers  biennial  conference 
of  economists  and  social  scientists,  the  Institute  for  Water  Resources  conducted  a 
workshop  on  uncertainty  in  benefit  analysis.  The  purpose  of  the  workshop  was  to  meet 
with  district  and  division  economists  to  discuss  details  of  the  IWR  work  unit, 
"Uncertainty  in  Benefit  Analysis."  Since  economists  only  convene  every  two  years,  it 
was  a  rare  occasion  to  meet  with  a  large  number  of  them.  It  also  allowed  researchers 
a  chance  to  discuss  the  project  definition  document  and  to  solicit  field  input  on 
where  the  research  effort  should  be  focused.  The  workshop  allowed  IWR  to  get  an 
extensive  response  to  a  questionnaire  and  have  an  open  discussion  of  the  most  critical 
issues  related  to  the  work  unit. 


WORKSHOP  AGENDA 

Tlie  workshop  consisted  of  a  formal  presentation,  a  discussion  of  major  research 
issues,  administration  of  the  questionnaire,  and  a  discussion  of  policy  issues.  The 
formal  presentation  included  a  review  of  the  project  definition  document,  which 
defined  the  purpose  and  scope  of  the  research  unit;  a  statement  of  the  requirements 
for  risk  and  uncertainty  analysis  in  Principles  and  Guidelines:  a  description  of  the 
classifications  for  sources  of  uncertainty;  and,  an  examination  of  the  various  methods 
for  dealing  with  uncertainty. 

The  presentation  was  followed  by  a  discussion  of  these  major  issues: 

1)  Study  Fund  Allocation:  How  are  funds  distributed  between  study  elements:  ie . 
economics,  hydrology  and  hydraulics,  and  environmental  branches?  How  can  economic 
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sections  allocate  funds  to  minimize  uncertainty?  How  can  resources  from  various  study 
elements  be  used  to  take  advantage  of  interdependent  needs? 

2)  Economists'  perceptions  of  the  major  sources  of  uncertainty. 

3)  Explicit  •Ways  We  Now  Handle  Uncertainty;  contingency  factors,  discount  rates, 
sensitivity  analysis,  and  limited  claiming  of  "future  benefits." 

4)  Implicit  Ways  We  Now  Handle  Uncertainty:  the  effects  uncertainty  may  have  on 
plan  formulation,  evaluation,  and  plan  selection. 

The  central  focus  of  the  workshop  was  the  administration  of  a  questionnaire.  The 
questionnaire  was  divided  into  seven  parts:  1)  respondents'  experience  with 
evaluating  flood  damage  reduction  project;  2)  Che  relative  Importance  of  the  benefits; 
3)  the  relative  magnitude  of  economic  analysis  as  compared  to  other  project  Casks;  4) 
Che  relative  costs  of  economic  analysis  as  compared  to  other  study  elements;  5)  study 
fund  allocation  for  economics  work;  6)  the  experiences  and  propensity  of  the 
economists  to  use  various  techniques  for  display  of  uncertainty;  and,  7)  economists' 
attitude  to  several  provocative  policy  questions. 

WORKSHOP  PARTICIPATION 

Copies  of  the  work  unit  project  definition  and  agenda  for  the  workshop  were 
mailed  out  in  advance  to  the  chief  of  economics  in  each  division  and  district. 
Participants  were  self-selected  by  making  the  commitment  to  stay  on  after  the 
economists  and  social  science  conference.  Twenty- five  district  and  division 
economists,  planners,  and  social  scientists  attended  the  workshop.  All  eleven  Corps 
divisions  were  represented.  Even  though  the  workshop  participants  represented  a  broad 
geographic  coverage,  it  should  not  be  assumed  that  those  attending  represented  a 
"scientific"  sample  of  the  population.  However,  some  of  the  patterns  of  response  to 
the  questionnaire  were  consistent  enough  to  suggest  that  they  are  fairly  strong 
indicators  of  the  collective  experience  and  attitudes. 


GENERAL  INSTRUCTIONS  FOR  THE  QUESTIONNAIRE 


The  following  instructions  were  given  in  the  introduction  to  the  questionnaire; 

As  was  mentioned  In  the  project  definition  document  for  the  "Uncertainty  in 
Benefit  Analysis”  work  unit,  we  are  attempting  to  assist  field  offices  in  addressing 
uncertainty  in  benefit  estimates .  Your  participation  in  this  workshop  session  is 
greatly  appreciated.  We  would  like  to  ask  for  your  continued  assistance  in  helping  us 
to  focus  our  research  so  that  the  needs  of  the  field  offices  are  adequately  addressed. 
The  following  questionnaire  has  been  developed  to  help  assess  the  manner  in  which 
uncertainty  has  been,  or  should  be,  addressed  in  estimating  the  benefits  of  flood 
damage  alleviation  projects.  As  a  result  of  the  emerging  nature  of  the  state-of-the- 
art  in  uncertainty  analysis,  and  as  a  result  of  our  attempt  to  not  make  the 
questionnaire  too  long,  complex  or  detailed,  some  of  the  questions  may  seem  vague  or 
overly  simplistic  with  regard  to  the  comprehensive  and  elaborate  nature  of  the 
activities  that  encompass  the  feasibility  study  process.  In  this  regard  it  is  useful 
to  keep  in  mind  that  we  are  seeking  your  expert  Judgement  and,  therefore,  would 
appreciate  your  making  an  informed  guess  on  some  of  these  issues.  We  have  provided 
space  on  most  questions  for  you  to  expand  yotur  answers,  or  to  provide  your  own 
categories,  comment  on  the  questions,  or  describe  any  difficulties  with  providing  an 
answer  to  the  questions. 

Ve  realize  that  there  is  a  wide  variety  of  flood  damage  project  components  that 
provide  alleviation  benefits.  To  structure  this  questionnaire  to  include  all  those 
possibilities  would  be  overly  prohibitive  in  time  and  resources.  Ve,  therefore,  ask 
that  your  answers  should,  to  the  extent  possible,  represent  an  amalgam  of  the 
experience  you  have  had  in  performing  or  reviewing  feasibility  studies  for  flood 
damage  alleviation  projects. 


RESULTS 

The  results  of  the  survey  are  recorded  and  Interpreted  below.  Four  statistics 
are  used  in  this  analysis:  the  mean  (average),  the  range,  the  standard  deviation,  and 
the  coefficient  of  variation.  The  standard  deviation  is  given  to  indicate  the  level 
of  dispersion  in  the  answers.  The  coefficient  of  variation  gives  the  level  of 
dispersion  relative  to  the  magnitude  of  the  mean. 

Note  that  the  questions  and  tabulation  of  responses  for  each  question  are  given 
in  bold  type . 
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QUESTIONS  AND  RESPONSES 


QUESTION  1.  RESPONDENTS'  EXPERIENCE  WITH  FLOOD  PROJECTS 

Please  indicate  below  the  percent  that  each  of  the  following  characteristics  and 
components  are  applicable  to  your  urban  flood  damage  reduction  feasibility  studies. 

The  mean  and  range  for  each  are  given  below; 


a. 

coastal  floodine  15.7 

riverine  floodine  38.2 

stream  flooding 

■  A6JL 

0-50 

P-?Q 

0-100 

b. 

sinele  numose  70.0 

multi-oumose  30.0 

Q-IQP 

0-100 

c . 

levees  and  flood  walls 

34.7  reservoirs  19.4 

channels 

43.8 

0-100  0-60 

P-90 

d. 

permanent  relocation  15 

.  9  flood  waming/response 

24.5 

floodnroofine  23.9 

(h 

0-100 

0-100 

Respondents  indicated,  as  we  expected,  a  vast  majority  work  on  inland  river  and 
stream  flooding  rather  than  coastal  flooding.  What  may  not  have  been  expected  10  or 
15  years  ago  is  a  major  shift  in  work  from  riverine  flooding  to  stream  flooding. 
There  also  appears  to  be  a  shift  away  from  multi-purpose  to  single -purpose  projects. 
This  has  occurred  as  the  Corps  has  been  building  fewer  reservoir  projects  and  become 
less  Involved  in  recreation.  Structural  measures  are  still  the  most  commonly 
considered  in  planning,  although  almost  25%  of  respondents  say  that  they  have  been 
involved  in  projects  where  flood  warning  was  considered  and  another  nearly  25%  say 
they  were  Involved  in  studies  where  floodproofing  was  considered. 
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QUESTION  2.  RELATIVE  IMPORTANCE  OF  BENEFIT  CATEGORIES 

Please  estimate  the  average,  minimum  and  maximum  percent  contribution  to  total 
benefits  from  each  of  the  following  categories.  Do  not  worry  about  average 
percentages  adding  to  100%.  We  are  only  Interested  In  relative  contributions. 

Average  Average 


Cateeorv 

Existing  physical  In  Oxidation 
reduction  benefits 

Average 

Percent 

67.1 

Coefficient 
of  Variation 
.24 

Minimum 

Percent 

44.3 

Maximum 

Percent 

83.0 

Future  physical  Inundation 
reduction  benefits 

11.5 

.61 

3.1 

20.5 

Existing  non-physical  Inundation 
reduction  benefits 

12.4 

.80 

1.8 

15.1 

Income  losses 

(1.4) 

.6 

8.1 

Emergency  costs 

(2.9) 

.9 

11.0 

Traffic  rerouting  and  delay 

(6.0) 

2.8 

17.5 

Floodproofing 

(1.7) 

,4 

14.8 

Administrative  costs  of  flood 
Insurance 

(1.7) 

2.0 

12.0 

Temporary  relocation  and 
reoccupation  costs 

(1.0) 

.6 

6.3 

Modified  use  of  floodproofing 
property 

(.6) 

.4 

4.0 

Restoration  of  land  market 
values 

(1.3) 

.9 

10.2 

Future  non-physical  Inundation 
reduction  benefits 

1.4 

2.71 

0 

6.5 

Location  benefits 

2.7 

1.19 

.3 

20.0 

Intensification  benefits 

3.5 

1.29 

.4 

16.4 

Advanced  bridge  replacement 

1.5 

1.33 

.3 

6.2 

Employment  Benefits 

1.8 

1.06 

0 

7.5 

Negative  benefits  (non-mltlgated 

-2.0 

2.6 

0 

-5.2 

Induced  damage) 

Other 

(Please  note  that  average  percents  for  all  major  categories  (not  including 
the  various  types  of  non-physical  benefits)  have  been  normalized  to  add  to 
100%.  The  components  of  non-physical  benefits  have  been  normalized  to  add 
to  12.4%,  the  sum  of  the  non-physical  benefit  contribution  to  the  total.) 


We  are  not  aware  of  any  standard  reporting  within  the  Corps  that  gives 
estimated  benefits  by  project  or  by  district.  The  large  cross-section  of 
individuals  represented  at  this  meeting  afforded  the  opportunity  to  start 
collecting  this  information.  If  we  could  not  obtain  statistically 
significant  estimates,  we  could  at  least  get  an  idea  of  the  relative 
importance  of  the  benefit  categories.  All  of  the  major  benefit  categories 
defined  in  Principles  and  Guidelines  for  Water  and  Related  Land  Resources 
Implementation  Studies  were  listed  in  this  question.  Respondents  were 
asked,  given  all  the  projects  that  they  have  worked  on  or  reviewed,  to 
Identify  the  average,  minimum,  and  maximum  of  total  project  benefits  that 
are  attributed  to  each  benefit  category.  The  relative  importance  of  each 
benefit  category  Indicates  the  significance  of  the  uncertainty  associated 
with  that  benefit  category. 

The  results  of  question  2  are  also  Illustrated  in  Figures  1  through  5. 
Figure  1  gives  the  average  proportion  each  of  the  major  benefit  categories 
contributes  to  the  total  benefits.  Figure  2  shows  how  the  average 
proportion  of  each  of  the  non-physical  benefit  categories  on  the  average 
contributes  to  total  non-physical  benefits.  Figures  3  through  5  show  the 
distribution  of  total  benefits  respectively  contributed  by  existing  physical 


and  non-physical  and  future  inundation  reduction  benefits. 
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UNCERTAINTY  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  2 


SllJ3N3a  3AliV03N 


Sil33N3g  iN3WA0^c^^N3  V 


iN3Ai30v3d3a  3oaiaa  aaoNVAOv^ 


SilJ3N38  NOIiVDUlSN3iNI 


Sil33N3a  NOliVOOl 


ivoiSAHd-NON  aanind  ;; 


3VOlSAHd-NON  9NliSIX3  ~ 


"ivoiSAHd  aaninj 


Siid3N3a  iviOi  01  NOiinaiaiNOO  %  oav 


FIGURE  I  -  BENEFIT  CATEGORIES  CONTRIBUTION  TO  TOTAL  BENEFITS 
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FIGURE  2  -  DISTRIBUTION  OF  PHYSICAL  CONTRIBUTION  TO  TOTAL  BENEFITS 
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EXISTING  NON-PHYSICAL  INNUNOATION  RED. 


nCURE  3  .  DisnuBunoN  of  non-physical  contribution  to  total  benefits 
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_ Questionnaire  results  -  ques.  2  subcat. 
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[ON  TO  TOTAL  B1 


QUESTION  3.  COMPONENT  CONTRIBUTION  TO  PHYSICAL  INUNDATION  REDUCTION 
BENEFITS 

Which  of  the  following  categories  generally  contribute  the  most  to 
total  physical  Inundation  reduction  benefits?  Please  Indicate  by  giving  the 
percentage  of  physical  damages  that  each  of  these  categories  typically 


compose . 

Residential  damage  45.4 
Commercial  damage  26 . 1 
Indvistrlal  damage  11.9 
Institutional  damage  6.2 


(e.g.  damage  to  government 
buildings,  hospitals,  churches, 
and  surrovindlng  property) 

Public  utility  damages  3.5 

Transportation  facility  damage  .  4.0 

Other  2 . 8 


Benefit  evaluation  of  many  project  reports  that  indicated  physical 
Inundation  reduction  benefits  have  traditionally  accounted  for  the  lion's 
share  of  benefits.  This  category  of  benefits  Is  so  large,  It  Is  important 
to  identify  the  components  of  the  category  to  have  a  meaningful  breakdown  of 
the  sources  of  benefits  that  are  most  Important.  Question  3  had  respondents 
rate  the  types  of  property  by  the  relative  contributions  to  the  overall 
project  benefits. 

The  average  contribution  of  each  major  type  of  property  to  total 
physical  inundation  reduction  benefits  is  Illustrated  in  Figure  6.  Figures 
7  through  9  give  the  respective  distribution  of  responses  for  residential, 
commercial,  and  Industrial  property.  The  figures  show  that  the  bulk  of 
inundation  reduction  benefits  come  from  residential  and  commercial  damages. 
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QUESnONNAIRe  RESUUS  -  QUESRON  3 


FIGURE  <  -  COMPONENTS  OF  PHYSICAL  INUNDATION  REDUCTION  BI 
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FIGURE  7  -  DISmUBUnON  FOR  RESIDENTIAL  SHARE  OF  PHYSICAL  COSTS 
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FIGURE  8  -  DlSTRIBUnON  FOR  COMMERCIAL  SHARE  OF  PHYSICAL  COSTS 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 


FIGURE  »  ■  DlCTRIBUnON  FOR  INDUSrrRIAL  SHARE  OF  PHYSICAL  COSTS 


QUESTION  4.  ALLOCATION  OF  STUDY  FUNDS 

The  allocation  of  study  funds  will  vary  by  district  and  project.  This 
question  was  included  to  get  an  estimate  of  the  average  allocation  to 
economics  relative  to  other  elements  of  the  study,  and  to  see  Just  how  much 
variation  there  might  be  between  those  averages.  It  is  an  indicator  of 
whether  funds  are  adequate  for  benefit  calculation  and  what  the  process  is 
to  establish  that  funding. 

The  survey  indicated  an  average  of  13.5%  of  study  funds  are  allocated 
to  economics,  with  a  range  of  5  to  50%,  and  a  median  value  of  12.5%.  There 
was  certainly  an  indication  that:  1)  cost  estimates  are  made  on  an 
individual  project  basis;  2)  standard  percentages  are  seldom  applied  to 
allocate  project  planning  funds;  and,  3)  economists  generally  make  their  own 
cost  estimates  and  then  negotiate. 


The  following  answers  were  given  as  to  how  study  funds  were  allocated: 


Economics  section  estimates  14 
Negotiated  between  project  elements  5 
Project  manager  assigns  values  6 
Determined  by  scale  of  the  project  2 
Funds  are  transferred  in  as  needed  1 
Appropriations  are  revised  after 

divi ?ion  review  of  report  1 
A  normal  allocation  is  based  on  past 

studies  1 
A  decision  is  made  by  higher  elements 

within  the  district  1 
Allocation  is  proportional  to  H+H  costs  1 
Don't  know  3 


Multiple  responses  are  recorded. 
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QUESTION  5.  ALLOCATION  OF  FUNDS  BY  STUDY  TASK 
In  general,  ho#  are  study  fissta  £oar  calculadxts  econcHlc  benefits  annrji»«t  ts  the  following  radm  In  estdaeCing 
flood  tedoddon  project  eocamLc  benedts? 


Awraga 

Coefficient 

Minilui 

Haxliiua 

Category 

of  Variation  ^epoe^ 

Bacgt 

Ebdstine  Condltlans 

a.  Delineate  Affected  Area 

8.7 

1.76 

4.1 

12.7 

and  Select  Beaches  for 

Analysis 

b.  In\«nCory  existing  floo(%>laln 

35.0 

.47 

-  esdiaate  elevations  of  structures 

35.0* 

.32 

8.3 

19.3 

-  detecDdne  yalue  of  structures 

25.(i* 

.32 

5.5 

20.4 

-  detenaine  value  of  oontarts 

19.0* 

.36 

2.3 

16.1 

-  detemlnB  value  of  outside  property 

11.0* 

.4 

1.6 

7.9 

-  other 

W.ti* 

.59 

0 

2.7 

c.  Detetidne  depth-daaage  lelatlonAlpe 

10.6 

1.07 

4.9 

18.7 

d.  CalfiiliifT  eaipected  nual  daiwges 

9.3 

.78 

2.7 

15.1 

Riture  Conditions 

e.  Project  population  snd  lari 

6.1 

.99 

2.7 

lO.S 

use  changes  In  affected  area 


f .  De'wlop  fiituEB  flootyiain  InwrCocy 


widiout-project: 

4.9 

1.27 

5.2 

1.3 

vLth-proJect: 

2.8 

1.14 

.6 

6.5 

g.  DetenalxiB  hituce  depth-doaigp 
relationdilps 

1.9 

1.52 

.3 

5.5 

h.  Calmlste  equivalent  anual  d— g** 

4.4 

1.09 

2.8 

6.7 

PtfTg  Pargaa 

1.  Calmlste  non-ptyslcal  benefits 

7.0 

.79 

2.6 

14.4 

j.  Calmlats  locarlno  orl 

Intensificatlan  benefits 

4.9 

1.27 

2.4 

19.4 

k.  Otlsr 

4.4 

1.77 

6.2 

17.0 

*  Indicates  a  percentage  of  total  Inventory  costs 
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Sixty- three  percent  of  resources  spent  in  flood  damage  reduction  studies 
were  spent  on  estimating  inundation  reduction  benefits  for  existing  conditions. 
Over  half  that,  35.0%  of  the  total,  was  spent  on  inventory  of  the  existing 
floodplain.  Aggregate  mean  values  and  the  coefficient  of  variation  on  the 
average  percent  of  resource  for  each  task  are  given  along  with  the  aggregate  mean 
values  for  minimum  and  maximum  of  economics'  resources  devoted  to  each  task.  The 
coefficients  of  variation  indicates  a  very  sizeable  difference  among  field 
offices  in  the  proportion  of  resources  devoted  to  each  task,  except  for  the  very 
basic  floodplain  inventory  work. 

Figure  10  gives  the  breakdown  of  the  average  allocation  of  resources  to  each 
of  the  major  task  in  benefit  computation.  Figures  11  through  16  give  the  average 
distribution  of  responses  for  each  of  the  major  tasks. 
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35  QUESTIONNAIRE  RESULTS  -  QUESTION  5 


yjHiO  V 

'N3a  JlSNSiNI  ^  iVOOl 
SilJ3N3a  iVOISAHd-NON 


S30Vl^va  IvnNNV  iN31VAin03  J 

'iONnj  30VAva-Hid3a  3dnind 


AdOiN3ANI  TOdd  HUM  3dninj 


AdOiN3ANI  0/m  iminj 


TOdd  3Sn  QNVl  3?  'dOd 

BSSSS 

S30vi^va  NNv  a3i33dx3  5 

ssssss 

V3dV  3lV3Nn3Q  ® 

- ! - : - ! - 1 - r- 

*— 

z 

o 

I 

fcO 

UJ 


z 

UJ 

OD 


CM 


CM 


Aaoo3iY3  01  saNHj  Aams  jo  nouvoohv  •s 


21 


FIGURE  10  -  PERCENT  OF  STTUDY  FUNDS  GOING  TO  EACH  MAJOR  TASK 
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*  ALLOCATION  OF  STUOV  FUNDS  TO  TASK 

figure  11  -  AVERAGE  ALLOCATION  OF  FUNDS  TO  DELINEATION  OF  STUDY 
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INVENTOR^r  EXISTIN6  RjOOOPUUN-qUES.  SB 


FIGURE  12  -  AVERAGE  ALLOCATION  OF  FUNDS  TO  FLOODPLAIN  INVENTORY 
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UNCERTAINTY  IN  BENEFIT  ANALYSIS 

CALC.  EXPECTED  ANNUAL  0AMA6ES-QUES.  5D 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 

FUTURE  CONDITIONS  -  SUM  (5E  THROUGH  5H) 


FIGURE  15  -  AVERAGE  ALLOCATION  OF  FUNDS  TO  TASKS  5E  THRU  5H 


QUESTION  6.  MAJOR  SOURCES  OF  UNCERTAINTY 


UiaC  do  yai  feel  are  the  aajor  aoutces  of  tzicertaijity  in  eadi  of  the  tadcs  in  question  4?  RjC  an  "X” 

vnd^  each  oolun  that  describes  a  source  of  uncertainty  for  that  task. 

Major  Sources  of  UrEectainty 


UnavailabllLty 


of  Reliable 
Insufficient  Faulty  Methods  of 

Tasks  Data  Data  Analvwfo _ 

(luher  of  responses) 


UhmricIpatBd  Chants 
In  Ctaii^rtcwv»  _ 


Other 

(Describe) 


RrisHrar  Conditions 

a.  DeHneatB  Affected  Area  10 

and  Select  Reaches  for 

Analysis 


b.  Inwntzay  eadstli^  fLood^plaln 

-  estdnata  structure  eLsvatlonB  13 

-  deteradne  value  of  structures  11 

-  detemline  value  of  rtw^ipnca  17 

-  detemlne  value  of  outside  property  11 

-  other  1 


6  2 


9  2 
11  2 
8  5 
6  3 
1  0 


7 


3 

6 

5 

3 

0 


1 


0 

0 

0 

0 

0 


c.  Deteodne  depth-dang^  relationEhlps  17 

d.  Calculate  expected  annual  denag^  3 
Riture  Conditions 

a.  Project  papulation  and  Issl  13 

use  changes  In  affected  area 

f .  Develop  future  floodplain 
lie^ntoiy: 


wlthout-project  12 

wlth-project  10 

g.  Determine  future  depth-damage  11 

relation^ilps 

h.  Calculate  esqiectBd  anial  A—pgag  4 

Other  Benefits 

1.  Calculate  non-ptyslcal  benefits  14 

j .  Calculate  location  and  14 

intensification  bene:0.ts 


14  6 

7  3 

5  8 

3  9 

3  8 

5  7 

9  3 

7  9 

6  6 


7 

9 

15 


17 

15 

7 

8 

6 

11 


0 

2 

0 


0 

0 

1 

1 

0 

1 


k.  Other 


0  0 


0 


0 


0 
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The  results  of  question  6  indicate  insufficient  and  faulty  data  are  particularly 
difficult  problems  throughout  the  planning  process,  while  unanticipated  conditions 
created  obstacles  to  accurate  estimates  of  future  population  and  future  floodplain 
inventory.  Insufficient  data  was  considered  to  be  the  greatest  overall  source  of 
uncertainty.  Faulty  data  was  a  particularly  bad  problem  for  estimating  structure 
values  and  elevations,  content  values,  depth-damage  relationships,  calculating 
expected  annual  damages,  and  benefits  for  alleviating  non-physical  costs. 

Respondents  were,  for  the  most  part,  satisfied  with  available  methods  for 
establishing  existing  conditions.  However,  about  one-third  of  the  respondents 
thought  that  current  methodology  was  inadequate  for  projecting  future  population  and 
land  use,  and  depth-damage  relationships. 

Responses  are  indicated  in  the  table  below.  In  addition,  figure  17  gives  the 
total  number  of  times  that  insufficient  data  was  identified  as  being  a  major  source  of 
uncertainty  for  each  of  the  major  tasks  in  benefit  computation.  Figures  18  through  20 
do  the  same  for  faulty  data,  unreliable  methods,  and  unanticipated  changes  in 
conditions . 
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MAJOR  SOURCES  OF  UNCERTAINTY 

INSUmCIENt  DATA 
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MAJOR  SOURCES  OF  UNCERTAINTY 

UNAVAILABIUTY  OF  RELIABLE  METHODS 
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MAJOR  SOURCES  OF  UNCERTAINTY 

UNANTICIPATED  CHANGES  IN  CONDITIONS 
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qUESnCN  7. 

R>r  dhe  calLi  Indlcata  with  <n  V  or  a  V  «■  CoUam: 

Mailc  with  an  V  -  if  thla  aoucoa  cf  martalncy  in  benafit  aada 
an  arlncyiata  Incinasa  In  atufy  funda  ml  tlaa. 

Btaa  sao  ba  sljpifIcaAly  raduoed  with 

Mazk  with  an  *N*  -  If  thla  aoucoa  of  imrtwir^y  in  benafilt  astla 
with  ahsCandal  Incxeaaea  in  study  Aada  and  tlaa. 

Btaa  cainfc  ba  al^^ilficarCly  zadLxxd  even 

-MUOS  SODECES  OF  OlICEBTlinr 


iisoFFiciEirr 

DATA 


B^UMCEBTIIIITT  Cil  BE  BEDOCEO 
I^DICEBTAIRT  CAIIOT  BE  BEOQCED 

DELIIEATE  ASEA  AID  SELECT  BEACHES 

IIVEHTOET  ElISTIIG  FLOOD  PLAII 
ESTIMATE  STBDCTOBE  ELETATIOIS 

DETEBMin  7 ALOE  OF  STBOCTOBES 

DETEBMin  7 ALOE  OF  COMTEITS 

DETEBMIIE  7AL0E  OF  OOTSIDEPROP. 

OTIEB 

lETElMllE  DEPTH-DAMAGE  BEUTIOISHIPS 
lALOCOUTE  EXPECTED  AIIOAL  DAMAGES 
’BOJECT  POP.  AID  LAID  OSE  CHAIGES 
fOTOBE  II7EITOBY  fITHOOT-PBOJECT 
fOTOlE  II7EIT0BT  IITH-PSOJECT 
'OTOBE  DEPTH-DAMAGE  BEUTIOISHIPS 
:ALC0UTE  EQ0I7ALEr  AIIOAL  DAMAGES 

:alcdlate  ioi-?hts:cal  beiefits 

ALC.  LOCATIOI  AID  IITEIS.  BEIEFITS 


OIAITICIPATED 
CHAIGES  II 
COIDITIOIS 


2  18 

2  13 

2  11 

2  4 
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Respondents  were  very  confident  that  the  uncertainty  inherent  in  insufficient  or 
faulty  data  could  be  significantly  reduced  for  every  task  in  benefit  calculation. 

There  was  less  confidence  concerning  uncertainty  due  to  unavailability  of  reliable 
methods.  Many  respondents  did  not  believe  that  more  reliable  methods  could  be 
developed  to  reduce  uncertainty  for  population  and  land  use  changes,  future  floodplain 
inventory,  calculation  of  non-physical  benefits,  or  location  and  intensification 
benefits.  Not  surprisingly,  there  was  a  good  deal  of  skepticism  that  uncertainty  due 
to  unanticipated  changes  could  be  reduced. 

Figure  21  illustrates  the  results  of  question  7  by  giving  the  total  number  of 
responses  for  all  respondents  and  tasks  on  the  reducibillty  of  uncertainty  due  to 
insufficient  data,  faulty  data,  unreliable  methods,  and  unanticipated  changes  in 
conditions.  The  cross- tabulation  of  questions  6  and  7  is  illustrated  in  Figure  22, 
where  the  distinction  between  reducible  and  non-reducible  uncertainty  is  made  only  for 
that  uncertainty  which  was  identified  as  significant  in  question  6. 
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questionnaire  results  -  QUESTION  7 


BY  SOURCE  OF  UNCERTAINTY 


UNCERTAINTY  IN  BENEFIT  ESTIMATION 

_  questionnaire  results  -  CROSS  6  AND  7 
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nCURE  22  -  CROSS-TABULATION  BETWEEN  QUECTKWS 


QUESTION  8.  USE  OF  UNCERTAINTY  ANALYSIS  METHODS 


a.  Ifav  frequently  ha\«  you  used  the  following  oethods  for  describing  aid 

measuring  uioertainty  in  benefit  estimates?  Please  indicate  by  placing  one 

letter  on  the  line. 

A  -  Vfe  haa  often  used  this  methods  in  our  studies 

B  -  Ue  haia  occasionally  used  this  method  in  our  studies. 

C  '  Ue  do  not  use  this  method  in  our  studies. 

_  Use  of  objactiia  statisclcal  measures  of  expected  values  and 

confidence  intervals. 

3A/11B/10C 

_  Use  of  SLfcjective  statistical  measures  of  expected  values  aid 

confidence  intervals. 

3A/12B/9C 

_  Use  of  subjective  expert  opinion  ^xxit  the  za^  of  values  for 

estimates  (l.e. ,  a  hi^,  mediin  and  low  estimate  of  wmnnir. 
growth) 

11A/11B/2C 

_  Use  of  sensitivity  aialysis  for  key  paraoetezs  and  variables  used 

In  the  benefit  estlmatinn  procedure. 

lOA/UB/lC 

_  Use  of  qualitati-ia  text  descriptions  of  the  types  ail  zelatiia 

magnitude  of  uncertainty  in  benefit  estimates. 

11A/11B/2C 

_  Use  of  graphic  displ^  tn  convey  the  types  and  relative  magnitudes 

of  mcertalnty  in  benefit  estimates. 

3A/11B/10C 
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b.  How  frequently  would  you  use  the  following  methods  for  describing  end 
measuring  vnrertainty  in  benefit  estdinacea?  Please  indicate  by  piecing  one 
letter  on  the  line. 


A  -  Ue  plan  to  use  this  method  &equently  in  &ture  feasibility  studies. 

B  -  Vfe  plan  to  use  this  method  occasionally  in  fixture  studies. 

C  -  Vie  do  not  hawe  ary  plans  to  use  dils  method  in  fixture  studies. 

_  Use  of  objectiw  statistical  measures  of  expected  values  and 

oonfidenoe  intervals. 

4A/14B/6C 

_ Use  of  sthjectlve  statistical  measures  of  expected  values  and 

confidence  intervals. 

6A/10B/8C 

_  Use  of  aihjectlw  expert  opinion  about  the  range  of  values  for 

estimates  (l.e. ,  a  hi^.  medlui  and  low  estimate  euxK«Hic 
growth) 

11  A/13B/0C 

_  Use  of  sensitivity  analysis  for  key  paiaoeters  and  variables  used 

in  the  benefit  estimation  procedure. 

17A/7B/0C 

_  Use  of  qualitative  text  descriptions  of  the  types  ml  xela£fi« 

mag^tude  of  ucertainty  in  benefit  estimates. 

13A/10B/1C 

_  Use  of  graphic  displays  to  oonmy  the  types  and  relative  magnitudes 

of  mcertainty  in  benefit  estimates. 

10A/10B/4C 
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Question  8  gives  some  of  the  most  common  techniques  used  to  identify 
and  display  uncertainty  in  benefit  analysis.  This  question  was  included  to 
determine  the  current  state  of  practice  and  the  intention  to  use  these 
techniques  in  the  future.  The  answers  to  this  question  are  given  above  and 
in  Figures  23  and  24.  The  results  indicate  a  definite  interest  in  the 
increased  use  of  several  methods  for  managing  uncertainty.  The  extent  of 
anticipated  change  in  use  for  each  method  is  illustrated  in  Figure  25.  Only 
the  use  of  statistical  measures  for  defining  confidence  limits  on  key 
variables  encountered  much  resistance.  It  should  be  noted,  however,  that 
those  confidence  Intervals  are  often  the  primary  Inputs  to  sensitivity 
analysis,  for  which  all  the  respondents  expressed  intention  to  use. 
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QUESTIONNAIRE  RESULTS  -  QUES.  BA 
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UNCERTAINTY  IN  BENEFIT  ESTIMATION 

QUESTIONNAIRE  RESULTS  -  QUES.  8B 


I 


42 


UNCERTAINTY  IN  BENEFIT  ESTIMATION 

QUESTIONNAIRE  RESULTS-QUES.  AA  AND  OB 


FIGURE  25  -  EXPECTED  CHANGE  IN  USE  FOR  METHODS  OF  DEALING 


QUESTION  9.  MOST  USEFUL  GUIDANCE 

Uiat  type  of  Infomatlcxi  would  be  nest  use&l  to  enphasize  in  a  oorual  on 
ixcertainty  In  benefit  a»lysls? 

Please  rate  on  scale  of  0  to  5,  O-not  useful,  S-ncst  usefiil. 

SOiXm) 
AVERAGE  DEVIATICN 


Descrlpticn  of  priinary  sources  of  ireertainty.  3.6  1.6 
Hew  to  allocate  project  finis  to  reduce  uncertainty.  2.3  1.8 
Techniques  for  estlnatii^  uncertainty.  4.7  0.5 
Techniques  for  dispL^  of  the  degree  of  uncertainty.  3.7  1.4 
Other:  3.8  3.4 


Respondents  clearly  and  consistently  indicated  that  the  most  useful 
guidance  would  be  on  techniques  for  estimating  uncertainty.  There  was  also 
strong  support  for  display  techniques  and  a  description  of  the  primary 
sources  of  uncertainty.  There  was  greatly  divided  opinion  on  the  usefulnes 
of  guidance  on  study  fund  allocation.  Other  types  of  guidance  suggested 
for  the  manual  included:  1)  case  studies  of  application  of  uncertainty  in 
benefit  analysis;  2)  typical  questions  asked  by  reviewers  from  the  Board  of 
Engineers  for  Rivers  and  Harbors  (BERH) ,  the  Office  of  the  Chief  of 
Engineers  (OCE) ,  the  Assistant  Secretary  of  the  Army  for  Civil  Works  (nSA) , 
and  the  Office  of  Management  and  Budget  (0MB);  3)  the  sensitivity  of  net 
benefits  to  various  types  of  uncertainty;  and,  4)  the  limitations  of 
uncertainty  analysis. 


QUESTION  10.  APPLICATIONS  OF  UNCERTAINTY  ANALYSIS 


Please  rate  the  degree  to  which  uncertainty  analysis  can  be  applied  to  other 
project  purposes.  Please  rate  on  a  scale  from  0  to  5,  0-not  applicable,  5- 
most  applicable. 


MEAN  STANDARD  DEVIATION 


Urban  flood  damage  reduction  4.3  1.0 
Agricultural  flood  damage  reduction  3.9  1.3 
Inland  Navigation  4.8  1.3 
Recreation  3.4  1.5 
Water  Supply  3.6  1.5 
Hydropower  3.2  1.8 
Deep  Draft  Navigation  4.6  .7 
Coastal  Flood  Protection  4.3 


Uncertainty  analysis  was  seen  as  very  applicable  to  all  types  of  the 
major  project  purposes  in  which  the  Corps  is  involved,  particularly  to 
navigation  and  flood  control  projects.  Uncertainty  analysis  was  also 
mentioned  as  being  applicable  to  erosion  control  projects. 
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QUESTION  11.  ATTITUDES  ON  POLICY  ISSUES 

For  Che  following  questions,  please  circle  the  number  which  indicates  the 
extent  you  agree  or  disagree. 


The  mean  and  standard  deviation  are  given  for  each  question. 

a.  With  present  techniques,  ve  carefully  account  for  the  benefits  from 
flood  damage  reduction  projects. 

strongly  -3-2-101  23  strongly 

disagree  agree 

Mean  -  1.3  Coefficient  of  Variation  —  1 

It  was  clear  from  the  answer  here  and  the  discussion  that  followed  that 
participants  were  at  least  moderately  satisfied  with  the  procedures 
available  for  evaluating  the  benefits  of  flood  damage  reduction. 


b.  Uncertainty  in  benefit  analysis  is  an  important  issue. 

strongly  -3  -2  -1  0  1  2  3  strongly 

disagree  agree 

Mean  -  1.7  Coefficient  of  Variation  -  .65 

There  was  strong,  nearly  unanimous  support  of  the  statement  that 
uncertainty  is  an  important  issue  in  benefit  analysis.  This  could  be  due  in 
part  to  the  fact  that  the  respondents  were  a  self-selected  group,  who  had  to 
stay  an  extra  day  for  the  workshop.  Those  that  were  in  the  strongest 
support  of  this  statement  said  that  the  economic  analysis  is  much  more 
useful  if  it  Identifies  the  degree  of  uncertainty  associated  with  any 
particular  benefits  and  the  degree  of  confidence  in  the  estimate  of  net 
benefits  for  each  alternative. 


c.  There  are  well  developed  techniques  for  estimating  uncertainty. 

strongly  -3  -2  -1  0  1  2  3  strongly 

disagree  agree 

Mean  —  -.9  Coefficient  of  Variation  —  2.11 

In  question  8  and  in  the  discussion  that  followed,  respondents 
indicated  that  they  used  techniques  for  estimating  uncertainty,  but  in  this 
question  it  appeared  that  they  did  not  necessarily  consider  the  techniques 
they  use  to  be  adequate.  They  appeared  to  be  looking  for  more  explicit 
techniques . 
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d.  The  fact  that  hydrologic  forecasting  nay  have  such  a  large  degree 
of  uncertainty  should  have  no  bearing  on  the  resources  devoted  to  benefit 
analysis . 

strongly  -3-2-10  123  strongly 

disagree  agree 

Mean  —  -.3  Coefficient  of  Variation  —  7.33 

Opinion  on  this  question  was  extremely  mixed,  with  strong  opinions  on 
both  sides.  Overall,  there  was  a  slight  edge  against  this  proposition.  It 
was  argued  that  uncertainty  about  hydrology  was  no  excuse  to  reduce  the 
precision  of  economic  analysis. 

e.  There  is  an  over-reliance  on  benefits  under  existing  conditions  for 
project  justification. 

strongly  -3  -2  -1  0  1  2  3  strongly 

disagree  agree 

Mean  —  - . 3  Coefficient  of  Variation  —  6 

There  was  a  slightly  negative  and  highly  fractionalized  vote  on  this 
issue.  Reviewers  routinely  look  more  closely  and  less  favorably  on  benefits 
claimed  under  future  conditions.  This  has  led  many  districts  to  spend  fewer 
resources  on  estimating  benefits  based  on  future  conditions  and  to  refrain 
from  counting  future  benefits  as  a  large  part  of  the  total  project  benefits. 

f.  The  degree  of  uncertainty  in  estimating  project  benefits  should  be 
a  major  parameter  in  project  selection. 

strongly  -3-2-10  1  2  3  strongly 

disagree  agree 

Mean  —  .2  Coefficient  of  Variation  —  9 

This  issue  had  respondents  equally  divided.  There  was  seldom 
a  neutral  opinion.  While  answering  question  9,  respondents  almost 
unanimously  advocated  the  value  of  techniques  to  measure  uncertainty;  they 
also  had  an  unfavorable  view  of  using  uncertainty  as  a  major  parameter  in 
project  selection.  Reasons  for  this  large  negative  reaction  were  due  in 
part  to  lack  of  confidence  in  techniques  currently  available  for  measuring 
uncertainty,  and  the  belief  that  we  can  still  be  confident  enough  in  our 
benefit-cost  estimates  to  accept  them  as  an  adequate  measure  of  economic 
efficiency. 


g.  A  25%  contingency  factor  should  be  added  to  the  total  benefits 
estimate  to  cover  benefits  "left  on  the  table". 

strongly  -3-2-1  0  1  2  3  strongly 

disagree  agree 

Mean  —  -.9  Coefficient  of  Variation  —  2.89 

This  statement  brought  the  strongest  and  most  consistent  negative 
reaction.  Wnile  several  people  said  that  there  are  probably  just  as  many 
benefits  unaccounted  for  as  cost,  and  that  it  would  be  only  fair  to  apply 
the  same  contingency  factor  to  benefits  as  costs,  the  majority  thought  that 
it  would  be  more  credible  and  generally  accepted  to  make  a  thorough  job  of 
computing  benefits. 


h.  Ex-post  studies  should  be  used  to  improve  the  accuracy  of  benefit 
estimate  methods. 

Mean  —  .9  Coefficient  of  Variation  —  4.11 

Several  people  argued  that  ex-post  studies  would  be  extremely  valuable 
for  tracing  the  economic  benefits  of  projects,  especially  for  projecting 
benefits  based  on  future  development.  There  are  reservations,  however, 
about  the  ability  to  isolate  the  effects  of  flood  protection  on  development 
from  what  otherwise  would  have  occurred  and  the  expense  of  conducting  the 
ex-post  studies. 

The  distributions  for  each  part  of  question  11  are  illustrated  in 
Figures  26  through  33. 
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UNCERTAINPr'  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  11A  15 


FIGURE  26  -  RESPONSES  TO  QUESTION  llA 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  11B 


FIGURE  27  -  RESPONSES  TO  QUESTION  IIB 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  11 C 
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FIGURE  28  -  RESPONSES  TO  QUESTION  IIC 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  11D 
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nCURE  29  -  RESPONSES  TO  QUESTION  IID 
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FIGURE  30  -  RESPONSES  TO  QUESTION  llE 
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UNCERTAINTY  IN  BENEFIT  ANALYSIS 

QUESTIONNAIRE  RESULTS  -  QUESTION  11C 


55 


ilGURE  32  -  RESPONSES  TO  QUECTION  IIG 


UNCERTAINTY  IN  BENEFIT  ANALYSIS 

_ WESnONUAIRE  RESULTS  -  QUESTION  11H 
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QUESTION  IIH 


CONCUISIONS 


Notwithstanding  the  limitation  of  sample  size,  the  questionnaire  was  found 
to  be  valuable  for  a  first  look  at  the  relative  importance  of  benefit  categories, 
getting  a  perspective  on  resource  allocation  for  economics  tasks,  getting  the 
economists'  perspective  of  the  major  sources  of  uncertainty,  and  determining 
current  attitudes  and  practices  of  economists  on  ways  of  handling  uncertainty  in 
benefit  analysis.  The  major  findings  of  the  questionnaire  are  as  follows; 

1)  This  group  conformed  to  the  assumption  that  the  greatest  part  of  the 
benefits  from  flood  damage  reduction  projects,  in  this  case  80%,  comes  from 
existing  inundation  reduction  benefits. 

2)  Likewise,  most  of  the  effort  for  estimating  benefits,  in  this  case  about 
63%,  goes  into  tasks  related  to  computing  existing  inundation  reduction  benefits. 

3)  The  economists  in  this  group  considered  insufficient  data  to  be  the 
greatest  source  of  uncertainty  in  benefit  analysis,  although  faulty  data, 
unreliable  methods,  and  unanticipated  changes  in  conditions  were  all  found  to  be 
significant  problems  for  various  stages  in  the  planning  process.  Most  of  the 
respondents  felt  that  there  was  potential  for  reducing  all  of  these  sources  of 
uncertainty. 

4)  A  large  portion  of  the  respondents  Indicated  that  they  were  already 
consistently  using  subjective  expert  opinion,  performing  sensitivity  analysis  for 
key  variables,  and  using  qualitative  descriptions  in  reports  to  handle 
uncertainty.  Use  of  all  these  methods  can  be  expected  to  increase,  and  most  of 
these  people  can  be  expected  to  consistently  use  graphic  display  of  uncertainty. 

Most  importantly,  this  questionnaire  and  the  workshop  have  given  direction 
to  the  uncertainty  in  benefit  analysis  research  effort.  The  continued  effort 
will  focus  on  details  of  the  primary  sources  of  uncertainty,  on  techniques  for 
illustrating  uncertainty,  and  on  techniques  to  display  uncertainty. 
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